Papers with bidirectional encoders
Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLU (2020.emnlp-main)
Copied to clipboard
| Challenge: | a number of languages are processed incrementally, but the best ones do not . we test five models on various datasets and compare their performance using three incremental evaluation metrics. |
| Approach: | They investigate how bidirectional LSTMs and Transformers behave under incremental interfaces . they propose to use bidirectional encoders in incremental mode while retaining non-incremental quality . |
| Outcome: | The proposed models perform better under incremental interfaces than the "omni-directional" BERT model, which achieves better non-incremental performance, but is impacted more by the incremental access. |
Efficient Encoders for Streaming Sequence Tagging (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing bidirectional encoders require a restart when a new token is received. |
| Approach: | They propose a Hybrid Encoder with Adaptive Restart that enables asynchronous encoding of a new token in an incremental streaming input. |
| Outcome: | The proposed encoder offers FLOP savings in streaming settings up to 71.1% and outperforms bidirectional encoders for streaming predictions by up to +0% streaming exact match. |
Garden Path Recovery in Causal and Masked Language Models (2026.acl-srw)
Copied to clipboard
| Challenge: | a linguistics study of garden-path sentences shows that recovery dynamics are important for linguistic evaluation . causal models show larger within-model disambiguation effects than masked models overall . |
| Approach: | They propose to compare garden-path recovery in causal and masked language models . they use 100 English garden- path/control pairs spanning three constructions . |
| Outcome: | The proposed model shows that decoder-only models exhibit sharper disruption at the point of syntactic revision, while encoders appear comparatively buffered at the disambiguator due to right-context access. |
When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementality (2024.acl-long)
Copied to clipboard
| Challenge: | In incremental models, one interpretation is possible, but models that can revise can do so if the ambiguity is resolved. |
| Approach: | They propose an interpretable way to analyse incremental states in a bidirectional way . they propose to use a model that can update internal states to reflect the garden path effect . |
| Outcome: | The proposed model shows that it can perform revisions and recover if the label is incorrect. |
Self-Modifying State Modeling for Simultaneous Machine Translation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for simultaneous machine translation fail to optimize the policy . existing methods require building a decision path to learn the policy, but they cannot explore all potential paths . |
| Approach: | They propose a new training paradigm that uses a read/write policy to optimize the policy . existing methods usually require building a decision path to learn a suitable policy a user makes . |
| Outcome: | The proposed model outperforms strong baselines and allows offline models to acquire SiMT ability with fine-tuning. |
BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for incorporating pre-trained models into NMT systems are non-trivial and lack a comparison of the impact that other pre-trainers may have on translation performance. |
| Approach: | They propose to use the input of a bilingual pre-trained language model as the input for NMT encoders and a stochastic layer selection approach to ensure sufficient utilization of contextualized embeddings. |
| Outcome: | The proposed bilingual pre-trained language model outperforms all other pre-train models on the IWSLT’14 dataset and the proposed dual-directional translation model. |